Assistance method for assisting performance of a task on a product, comprising displaying a highlighting image highlighting a monitored part of the product

ABSTRACT

This assistance method is intended for assisting an operator in performing a task on a product ( 12 ). To that end, the method comprises a step of displaying, superimposed on a view of a scene including a monitored part of the product ( 12 ) on which a subtask of the task has to be performed, at least one assistance image ( 100, 136 ) intended to assist the operator in performing this subtask. Said at least one assistance image ( 100, 136 ) includes at least one highlighting image ( 100, 102 ) highlighting the monitored part.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage application ofPCT/IB2017/001635, filed Nov. 21, 2017, which published on May 31, 2019as WO 2019/102245 A1, all of which is hereby incorporated by referencein its entirety.

The present invention concerns an assistance method for assisting anoperator in performing a task on a product, of the type comprising astep of displaying, superimposed on a view of a scene including amonitored part of the product on which a subtask of the task has to beperformed, at least one assistance image intended to assist the operatorin performing this subtask.

The present invention also concerns an assistance system for assistingan operator in performing a task on a product, the assistance systemcomprising a display management unit configured for displaying,superimposed on a view of a scene including a monitored part of theproduct on which a subtask of the task has to be performed, at least oneassistance image intended to assist the operator in performing thissubtask.

It is common that, after a product has been produced, said product hasto undergo a quality control to ensure that the product meets some setspecifications prior to further operations are performed on the productor before the product is sent to a client. Such a quality controlexists, for instance, for controlling the quality of automotive vehicleengines.

Usually, such a quality control is executed by a human operator who isrequired to perform a list of controls on the product, each controlconsisting of verifying that a part of the product meets somecorresponding criteria, and to report a defect in case at least one ofthe controlled parts does not meet the corresponding criteria. When theproduct belongs to a range of products including several variants, thelist of controls depends habitually on the variant to which the productcorresponds.

Generally, the list of controls is provided as a paper printed form thatsummarizes each control in a few words, and the operator writes theresults of his controls on said form. This has several drawbacks:

-   -   this form is not intuitive and requires the operator to have a        lot of complementary knowledge to be able to perform the        controls correctly, so that a large amount of training hours is        needed for an already skilled assembly operator to be trained;    -   the operator has the tendency to rely too much on his memory and        skills and, as his cognitive load is high, this leads to a risk        that he does not perform the control correctly and lets some        errors and defect pass the controls;    -   there is a risk that the operator misses certain control lines,        thus resulting in the operator not controlling some defective        parts and letting some errors and defect pass the controls; and    -   the results of the controls need to entered in a digital form on        the day after by other operators, which leads to time loss        because of this necessary double data entry and delays detection        of a defective production station.

In response to these issues, there has been proposed a first solutionconsisting of using buttons as quality control reporting means. However,this first solution does not help the operator to know how he mustoperate his controls depending on the variant to which the productcorresponds, so that the cognitive load remains high for the operator,lots of complementary knowledge is still needed for this operator, andthe risk of errors remains high.

There has also been proposed a second solution using augmented reality,wherein information is displayed on a display surface superimposed on aview of scene including the controlled product. However, in this secondsolution, positioning of the information on the display surface isindependent from the position of the controlled product in the view ofthe scene; and this information does not allow the operator to knowwhere he should put his attention on the product to perform eachcontrol. As a result therefrom, the cognitive load remains high for theoperator, lots of complementary knowledge is still needed for thisoperator, and the risk of errors remains high.

An objective of the invention consists of assisting an operator inperforming a task on a product, so as to reduce the risk of errors fromthe operator while performing the task. Specific objectives of theinvention consist in reducing the cognitive load of the operator whileperforming the task, reduce the needed knowledge for the operator to beable to complete the task, and prevent the operator from missing somesubtasks while executing the task. Another objective consists offacilitating report of the accomplishment of the task by the operator.

To that end, the invention relates to an assistance method of the typementioned above, wherein the at least one assistance image includes atleast one highlighting image highlighting the monitored part.

According to specific embodiments of the invention, the assistancemethod also comprises one or more of the following features, consideredalone or according to any technically possible combination(s):

-   -   the at least one highlighting image comprises a ring surrounding        the monitored part and preferably centered on said monitored        part, and/or a virtual representation of said monitored part,        said virtual representation being positioned so as to        substantially match the monitored part in the view of the scene,    -   the at least one assistance image is displayed on at least one        display surface,    -   the view of the scene consists of a view of the scene through        the at least one display surface,    -   the at least one display surface is part of a head-mounted        display device, said head-mounted display device being adapted        to be worn by the operator in such a manner that the or each        display surface faces a respective one of his eyes,    -   the at least one display surface includes a primary display        surface and a secondary display surface, the at least one        assistance image consisting of a stereoscopic image including a        primary image displayed on the primary display surface and a        secondary image displayed on the secondary display surface, said        primary and secondary images consisting of images of a same        element viewed from slightly different angles,    -   the view of the scene is provided by a view source, and the        method further comprises the following steps:        -   tracking a displacement of the view source from a first            viewing position in the scene, in which the product occupies            a first relative position with a first relative orientation            relatively to the view source, toward a second viewing            position in which the product occupies a second relative            position with a second relative orientation relatively to            the view source,        -   calculating an estimation of the second relative position            and of the second relative orientation from the first            relative position and orientation and from the tracked            displacement, and        -   refreshing the display of the at least one assistance image            so as to adjust said at least one assistance image to the            estimation of the second relative position and orientation,    -   the view source comprises the position intended to be occupied        by at least one of the eyes of the operator wearing the        head-mounted display device,    -   the first viewing position consists of an initial viewing        position in which the first relative position and orientation        consist of predetermined relative position and orientation known        by a display management unit in charge of displaying the at        least one assistance image,    -   the assistance method comprises a step of equipping the product        with a support comprising a slot able for receiving the view        source and adapted so that, when the support is equipped on the        product and the view source is received in said slot, the view        source is placed in said initial viewing position, followed by a        step of placing the view source in the slot of said support,    -   the at least one assistance image comprises at least one        guidance image highlighting an area where to place the view        source so as to provide a better view on the monitored part,    -   the assistance method comprises a step of displacing the view        source in the scene in manner that the distance of the view        source from the monitored part crosses a threshold distance,        followed by a step of modifying the position of at least part of        the at least one assistance image,    -   the task to be performed comprises, and in particular consists        of, a quality control of the product,    -   the product comprises, and in particular consists of, a part of        an automotive vehicle, such as an engine of an automotive        vehicle,    -   the assistance method comprises, prior to displaying the at        least one assistance image, a step of automatically reading an        identification code associated to the product, followed by a        step of, using said identification code, automatically        retrieving a list of at least one subtask to be performed on the        product, the at least one assistance image being intended to        assist the operator in performing a subtask of said list.

The invention also relates to an assistance system of the type mentionedabove, wherein the at least one assistance image includes at least onehighlighting image highlighting the monitored part.

According to specific embodiments of the invention, the assistancesystem also comprises one or more of the following features, consideredalone or according to any technically possible combination(s):

-   -   the at least one highlighting image comprises a ring surrounding        the monitored part and preferably centered on said monitored        part, and/or a virtual representation of said monitored part,        said virtual representation being positioned so as to        substantially match the monitored part in the view of the scene;    -   the assistance system comprises at least one display surface,        the display management unit being configured to display the at        least one assistance image on said at least one display surface;    -   the view of the scene consists of a view of the scene through        the at least one display surface;    -   the assistance system comprises a head-mounted display device        which includes the at least one display surface, said        head-mounted display device being adapted to be worn by the        operator in such a manner that the or each display surface faces        a respective one of his eyes;    -   the at least one display surface comprises a primary display        surface and a secondary display surface, and the display        management unit is configured for displaying a primary image on        the primary display surface and a secondary image on the        secondary display surface, said primary and secondary images        consisting of images of a same element viewed from slightly        different angles, so as to form the at least one assistance        image as a stereoscopic image resulting from the combination, by        the eyes of the operator, of the primary and secondary images;    -   the assistance system includes a view source, at least one        sensor to track a displacement of the view source from a first        viewing position in the scene, in which the product occupies a        first relative position and a first relative orientation        relatively to the view source, toward a second viewing position        in which the product occupies a second relative position and a        second orientation relatively to the view source, and the        display management unit is configured to calculate an estimation        of the second relative position and orientation from the first        relative position and orientation and from the tracked        displacement, and to refresh the display of the at least one        assistance image so as to adjust said at least one assistance        image to the estimation of the second relative position and        orientation;    -   the view source comprises the position intended to be occupied        by at least one of the eyes of the operator wearing the        head-mounted display device;    -   the assistance system comprises a support able to be mounted on        the product and including a slot able for receiving the view        source, said support being adapted so that, when it is mounted        on the product and the view source is received in said slot, the        view source is placed in the initial viewing position;    -   the first viewing position consists of an initial viewing        position in which the first relative position and orientation        consist of predetermined relative position and orientation known        by the display management unit;    -   the at least one assistance image comprises at least one        guidance image highlighting an area where to place the view        source so as to provide a better view on the monitored part;    -   the display management unit is configured for modifying the        position of at least part of the at least one assistance image        when the distance of the view source from the monitored part        crosses a threshold distance;    -   the task to be performed comprises, and in particular consists        of, a quality control of the product;    -   the product comprises, and in particular consists of, a part of        an automotive vehicle, such as an engine of an automotive        vehicle;    -   the system comprises a memory storing a virtual mockup of the        product;    -   the system comprises a memory storing a list of at least one        subtask to be performed, a reader able to read an identification        code worn by the product, and a configuration unit configured to        retrieve said list in the memory, using the identification code        read by the reader, and the display management unit being        configured to display the at least one assistance image        depending on the relevant list retrieved from the memory.

Other features and advantages of the invention will appear upon readingthe following description, provided solely as an example and done inreference to the appended drawings, in which:

FIG. 1 is a schematic view of an assistance system according to theinvention,

FIG. 2 is a perspective view of a head-mounted display device of theassistance system of FIG. 1,

FIG. 3 is an exemplary view of a first display provided by theassistance system of FIG. 1, superimposed on a first view of a sceneincluding the product on which the task has to be performed,

FIG. 4 is an exemplary view of a second display provided by theassistance system of FIG. 1, superimposed on a second view of the scene,

FIG. 5 is an exemplary view of a third display provided by theassistance system of FIG. 1, superimposed on a third view of the scene,

FIG. 6 is an exemplary view of a fourth display provided by theassistance system of FIG. 1, superimposed on the third view of thescene,

FIG. 7 is an exemplary view of a fifth display provided by theassistance system of FIG. 1, superimposed on the third view of thescene,

FIG. 8 is a block diagram of a first part of an assistance methodaccording to the invention, implemented with the assistance system ofFIG. 1, and

FIG. 9 is a block diagram of a second part of the assistance method ofFIG. 8.

The assistance system 10 shown in FIG. 1 is intended to assist anoperator in performing in task on a product 12 (see FIG. 3). The task tobe performed consists in particular of quality control of the product12, the product consisting typically of an engine of an automotivevehicle such as a truck (not shown).

To that end, the assistance system 10 comprises a front-end displaydevice 14, a back-end server 16, and a support 18.

With reference to FIG. 2, the front-end display device 14 consists of ahead-mounted display device (referred to as “HMD device” in FIGS. 8 and9) comprising a frame 20 adapted to be mounted on the head of a user,and an interactive device 22 configured for interacting with a userhaving the frame 20 positioned on his head.

The frame 20 comprises a front beam 24 and two arms 26 projecting fromthe front beam 24 toward a rear end of the head-mounted display device14, the front beam 24 and the arms 26 encircling together a free area 28intended for reception of the head of the user.

The interactive device 22 comprises a primary display surface 30, asecondary display surface 32, a display management unit 34 to displayimages on the display surfaces 30, 32, a displacement tracking unit 36to track displacement of the head mounted device 14 relatively to areference position and a reference orientation thereof, at least onemicrophone 38 to input speech from the user, and at least one audiospeaker 40 to output sound to the user.

The primary and secondary display surfaces 30, 32 are placed on thefront beam 24 in such a manner that, when the head-mounted displaydevice 14 is worn by a user, the primary display surface 30 faces afirst eye of the user and the secondary display surface 32 faces thesecond eye of the user.

Each one of these display surfaces 30, 32 is transparent, so that theuser wearing the head-mounted display device 14 can see a view of ascene through said display surface 30, 32. The positions (not shown)intended to be occupied by the eyes of the user wearing the head-mounteddisplay device 14 thus form a view source from which a view of a sceneis provided.

The display management unit 34 comprises means 42, 44 to display imageson the display surfaces 30, 32, without hindering sight of the userthrough the display surfaces 30, 32. These means 42, 44 typicallyconsist of a primary projector 42 to project the displayed images on theprimary display surface 30, and of a secondary projector 44 to projectthe displayed images on the secondary display surface 32.

The display management unit 34 furthermore comprises a circuitry (notshown) to pilot the projectors 42, 44 and calculate the images to bedisplayed on the display surfaces 30, 32. This circuitry is preferablyhoused in the front beam 24 and includes, for example, at least oneprocessor and at least one memory.

In particular, the circuitry is configured for piloting the primaryprojector 42 so that said primary projector 42 displays a primary imageon the primary display surface 30 and for piloting the secondaryprojector 44 so that said secondary projector 44 displays a secondaryimage on the secondary display surface 32, said primary and secondaryimages consisting of images of a same element viewed from slightlydifferent angles and adapted so as to form at least one stereoscopicimage resulting from the combination, by the eyes of the user, of theprimary and secondary images.

The display management unit 34 is configurable so as to have knowledgeof the positions occupied by the eyes of a user wearing the head-mounteddisplay device 14, relatively to the display surfaces 30, 32. In otherwords, the display management unit 34 is configurable so as to haveknowledge of the position of the view source relatively to thehead-mounted display device. This allows the display management unit 34to place properly the at least one stereoscopic image in the view of thescene.

The displacement tracking unit 36 comprises a plurality of positioningand orientation sensors 50, 52, 54 to track displacement of thehead-mounted display device 14.

These sensors 50, 52, 54 are preferably housed in the front beam 24, asshown here. They include at least one visible-spectrum tracking camera50 for use in capturing images of surrounding surfaces (said cameras 50being, in the provided example, in the number of four, two of whichbeing positioned on a left side of the front beam 24, the two othersbeing positioned on the right side of the front beam 24), at least oneinfrared spectrum depth camera 52 for use in determining distances tonearby surfaces (e.g., for use in surface reconstruction to model theuser's environment), coupled to an infrared illumination source (notshown), and an inertial measurement unit 54 to measure accelerations androtations of the head-mounted display device 14.

The displacement tracking unit 36 further comprises a calculator (notshown) to receive the data collected by the sensors 50, 52, 54, deducetherefrom an estimation of the position and of the orientation of thehead-mounted display device 14 in real-world space, compare thisestimation with a reference position and a reference orientation, anddeduce therefrom a displacement of the head-mounted display device 14relatively to said reference position and reference orientation. Thiscalculator is preferably housed in the front beam 24 and includes, forexample, at least one processor and at least one memory.

The displacement tracking unit 36 is configured to provide thecalculated estimation to the display management unit 34.

The head-mounted display device 14 further comprises at least onevisible-spectrum standard camera 56 for use in capturing standard videoof what the user sees. This standard camera 56 is preferably, as shown,housed in the front beam 24.

In addition, the head-mounted display device 14 comprises a datacomputing unit 57 which is preferably, as shown, housed in the frontbeam 24. This data computing unit 57 comprises a memory (not shown) anda processor (not shown) associated to said memory, the memory storing atleast one software program adapted to be executed by the processor.

The head-mounted display device 14 also comprises a communication unit58, in particular a wireless communication unit, for communication ofthe head-mounted display device 14 with the back-end server 16. Asshown, this communication unit 58 is preferably housed within the frontbeam 24.

The head-mounted display device 14 furthermore comprises a headband 60to improve fitting of the head-mounted display device 14 on the head ofthe user. This headband 60 is encircled by the frame 20 and surroundsthe free space 28.

The head-mounted display device 14 typically consists of a Microsoft®HoloLens® device.

The back-end server 16 comprises a communication module 62, inparticular a wireless communication module, for communication of theback-end server 16 with the head-mounted display device 14. To that end,the communication module 62 uses the same communication protocols as thecommunication unit 58 of the head-mounted display device 14.

The back-end server 16 further comprises a memory 64 storing, associatedto an identifier of the product 12:

-   -   a virtual, three-dimensional mockup of said product 12, said        virtual mockup consisting preferably of an exact digitalized        three-dimensional mockup of the product 12, as created by the        department in charge of the development of the product 12,        transformed into a light weight format, preferably a format in        which the mockup has 100 000 facets or less,    -   an initial relative positioning information, said initial        relative positioning information consisting typically of a        default viewing position and orientation of the virtual mockup,        and    -   a list of subtasks to be performed on the product 12 for the        execution of the task.

Preferably, the memory 64 also stores at least one other mockup, atleast one other initial relative positioning information, and at leastone other list of subtasks, said other mockup(s), initial relativepositioning information and list(s) being associated to otheridentifiers than the identifier of the product 12.

The support 18 comprises a bracket 70 adapted to be mounted on theproduct 12 at a predetermined position on said product 12, an arm 72projecting from said bracket 70, and a slot 74 for receiving thefront-end display device 14, the slot 74 being disposed at an end of thearm 72 opposite the end by which the arm 72 is connected to the bracket70. The support 18 is adapted so that, when it is mounted on the product12 and the front-end display device 14 is received in the slot 74, thenthe front-end display device 14 is placed in an initial viewing positionrelatively to the product 12.

The support 18 is for instance 3D-printed.

According to the invention, the assistance system 10 is configured sothat the display management unit 34 may display, superimposed on theview of the scene provided from the view source, at least one assistanceimage 100, . . . , 136 (FIGS. 3 to 6) intended to assist the operator inperforming his task, said at least one assistance image 100, . . . , 136consisting of a stereoscopic image obtained by the combination of theprimary image displayed on the primary display surface 30 and of thesecondary image displayed on the secondary display surface 32.

To that end, the head-mounted display device 14 is provided with a codereader and a configuration unit.

The code reader is typically provided in the form of a software programstored in the memory of the data computing unit 57 of the head-mounteddisplay device 14 and adapted to be executed by the processor of saiddata computing unit 57.

The code reader is configured for receiving an image captured by thestandard camera 56, for recognizing in said image an identification codehaving a predetermined format, and for decoding said identification codeinto an identifier.

The configuration unit is typically provided in the form of a softwareprogram stored in the memory of the data computing unit 57 of thehead-mounted display device 14 and adapted to be executed by theprocessor of said data computing unit 57.

This configuration unit is configured to retrieve in the memory 64 ofthe server 16, using the identifier decoded by the code reader, thevirtual mockup, the initial relative positioning information, and thelist of subtasks which are, in said memory 64, associated to saididentifier. The configuration unit is further configured to retrieve,for each subtask within said list of subtasks, a part information, anaction information and, optionally, at least one of a picture and ananimation corresponding to said subtask, and to provide the virtualmockup and the list of subtasks to the display management unit 34together with said part information, action information, picture and/oranimation.

The part information designates the virtual representation, in thevirtual mockup, of a monitored part of the product 12 on which thesubtask has to be performed.

The action information identifies an action which has to be executed onthe monitored part to perform the subtask; for instance, this actionconsists of one of taking a look at the monitored part and of testingability of the monitored part to be displaced.

The picture consists typically of a picture showing the monitored partin a state in which said part is in compliance with the quality controlrequirements.

The animation consists typically of a video showing how the actionshould be performed.

The display management unit 34 is configured to display the at least oneassistance image 100, . . . , 136 depending on the virtual mockup, thepart information, the action information, the picture and/or theanimation provided to it by the configuration unit.

With reference to FIG. 3, the display management unit 34 is inparticular configured to display at least one assistance image 100, 102consisting of a highlighting image highlighting the monitored part. By“highlighting the monitored part”, it is meant that the at least onehighlighting image 100, 102 is able to draw the attention of theoperator wearing the head-mounted display device 14 on the monitoredpart. To that end, the at least one highlighting image 100, 102 ispositioned in the view of the scene so as to be in a specificrelationship with the position of the monitored part in said view; inother words, the at least one highlighting image 100, 102 has apredetermined positioning relative to the monitored part. The at leastone highlighting image 100, 102 is for instance centered or superimposedon said monitored part.

These highlighting images 100, 102 are here in the number of two andcomprise a ring 100 surrounding the monitored part and in particularcentered on said monitored part, and a virtual representation 102 ofsaid monitored part, said virtual representation 102 being positioned soas to substantially match the monitored part in the view of the scene.Alternatively, the at least one assistance image 100, . . . , 136comprises a single highlighting image 100, 102 consisting of one of thering 100 and of the virtual representation 102.

The virtual representation 102 consists typically of the virtualrepresentation of the monitored part, extracted by the display unit 34from the virtual mockup of the product 12 on the basis of the partinformation provided by the configuration unit.

The display management unit 34 is also configured to display at leastone assistance image 104, 106 consisting of a guidance imagehighlighting an area where to place the view source so as to provide abetter view of the monitored part. This at least one guidance image 104,106 is intended so that the operator may perform the full task by goingaround the product 12 only once, thus minimizing the overall number ofsteps of the operator needed for the execution of the task, andtherefore reducing the time needed for completing this task.

To that end, these guidance images 104, 106 are here in the number oftwo and comprise a square 104 superimposed on a region of the floorwhere the operator should stand to be able to see the monitored part,and a succession of arrows 106 pointing toward said region of the floor.Alternatively, the at least one assistance image 100, 136 comprises asingle guidance image 104, 106 consisting of one of the square 104 andof the succession of arrows 106. Alternatively, the succession of arrows106 is replaced by a single arrow (not shown).

The display management unit 34 is configured to display said at leastone guidance image 104, 106 when the monitored part is not visible inthe view of the scene.

With reference to FIG. 4, the display management unit 34 is furtherconfigured to display at least one assistance image 108, 110 consistingof an action explanation image intended for explaining to the operatorthe action to be performed.

Here, the action explanation images are in a number of two and comprisea shield 108 wearing a pictogram depicting the action to be executed onthe monitored part, and a frame 110 presenting the picture and/or theanimation provided by the configuration unit.

The display management unit 34 is configured to display the shield 108so that it faces a position in the scene where the view on the monitoredpart is optimal. The pictogram depends on the action informationprovided by the configuration unit; for instance, the pictogram consistsin two opposed right arrows, as shown, when the action consists oftesting the ability of the monitored part to be displaced by pushing andpulling said monitored part, in two opposed curved arrows (not shown)when the action consists of testing the ability of the monitored part tobe displaced by rotating the monitored part, or in an eye when theaction merely consists of taking a look at the monitored part.

The display management unit 34 is configured to position the shield 108adjacent to the ring 100, preferably in substantially the same plane assaid ring 100.

In addition, the display management unit 34 is configured to displayassistance images 112, 114 consisting of a validation mark 112 and adefect mark 114, said marks 112, 114 being preferably positioned, asshown, adjacent to the shield 108, in substantially the same plane assaid shield 108, the validation mark 112 being typically placed abovethe shield 108 and the defect mark 114 being typically placed under theshield 108.

Furthermore, the display management unit 34 is configured to display anassistance image 116 consisting of a task advancement image, intended toinform the operator about his advancement in the achievement of thetask.

This task advancement image 116 comprises here an ordinal 118 of thesubtask which is in the status of being executed by the operator; thisordinal 118 is advantageously, as shown, highlighted. The taskadvancement image 116 further comprises information 120 about remainingsubtasks which are still to be performed before achievement of the task,this information being here provided by the ordinals 122, 124 of saidremaining subtasks, said ordinals 122, 124 not being highlighted andbeing positioned on a first side of the ordinal 118. Advantageously, thetask advancement image 116 also comprises information 126 about thestatus of finished subtasks; this information is advantageouslypositioned, as shown, on a second side of the ordinal 118 opposite thefirst side, and is preferably provided, as shown, as a validation mark128 for the subtasks in which the operator has validated compliance ofthe controlled part with the quality control requirements, and as adefect mark 130 for the subtasks in which the operator has signalednon-compliance of the controlled part with the quality controlrequirements, said marks 128, 130 being here arranged in the same orderas the order in which these subtasks have been executed.

The display management unit 34 is in particular configured to displaythe task advancement image 116, when the view source is at a distance ofthe monitored part above a threshold distance, as a carrousel floatingabove the product 12, as shown in FIGS. 3 and 4, said carrousel beingoriented so that the ordinal 118 faces the view source, and to displaythe task advancement image 116 as a succession of icons adjacent to thering 100 and placed substantially in the same plane as said ring 100, asshown in FIG. 5, when the view source is at a distance of the monitoredpart under said threshold distance. Thus, the display management unit 34is configured for modifying the position of the task advancement image116 when the distance of the view source from the monitored part crossesthe threshold distance.

Moreover, the display management unit 34 is configured to display anassistance image 132 consisting of a statistics button for access to thestatistics of the operator. This statistics button 132 is preferably, asshown, integrated in the carrousel when the view source is at a distanceof the product 12 above the threshold distance, and positioned adjacentto the ring 100 when the view source is at a distance of the product 12under the threshold distance.

Besides, the display management unit 34 is configured to display anassistance image 134 consisting of a statistics menu, as shown in FIG.6, when a corresponding instruction has been provided by the operator,for instance as a vocal instruction, or as a movement of the handshowing the intention of the operator to tap the statistics button 132.

The display management unit 34 is configured to display the statisticsmenu 134 as a window superimposed on the view of the scene and showingthe number of control hours worked by the operator, the number ofdetected defects, and the number of validated products.

Additionally, the display management unit 34 is configured to display anassistance image 136 consisting of a defects menu, as shown in FIG. 7,when an instruction has been provided by the operator that the monitoredpart is defective, for instance as a vocal instruction, or as a movementof the hand showing the intention of the operator to tap the defect mark114, or as a combination of a vocal instruction and of a specificmovement of a hand of the operator.

The display management unit 34 is configured to display the defects menu136 as a window superimposed on the view of the scene and listing a listof defect causes, including for instance:

-   -   a wrong assembly,    -   a missing part,    -   a broken part,    -   an interference or a clash,    -   a wrong orientation,    -   a leakage, and    -   a wrong part.

So as to position the at least one assistance image 100, . . . , 136properly in the view of the scene, and so as to display the properassistance images 100, . . . , 136 depending on the position and theorientation of the view source, the display management unit 34 isconfigured to calculate an estimation of a relative position andorientation of the view source relatively to the product 12, and toadjust the at least one assistance image 100, . . . , 136 depending onthe estimation of said relative position and orientation. The displaymanagement unit 34 is further configured to update the calculation ofthe estimated position and orientation at a predetermined frequency, andto refresh the display of the at least one assistance image 100, . . . ,136 at the same frequency so as to continuously adjust said at least oneassistance image 100, . . . , 136 to said estimated relative positionand orientation. Preferably, this frequency is equal or superior to 60Hz.

To calculate the estimation of the relative position and orientation ofthe view source relatively to the product 12, the displacement trackingunit 36 is configured to record the estimation of the position and ofthe orientation of the head-mounted display device 14 in real-worldspace as the reference position and the reference orientation, and thedisplay management unit 34 is configured to calculate the estimation ofthe relative position and orientation from the predetermined relativepositioning information provided by the configuration unit and from thecalculated displacement provided by the displacement tracking unit 36.

The assistance system 10 is further configured so as to record reportingof the operator on his execution of the task. To that end, thehead-mounted display device 14 is provided with a reporting unit.

This reporting unit is typically provided in the form of a softwareprogram stored in the memory of the data computing unit 57 of thehead-mounted display device 14 and adapted to be executed by theprocessor of said data computing unit 57.

The reporting unit is configured to record a status of each subtasklisted in the list of subtasks retrieved by the configuration unit, thisstatus being either a validated status of a defective status.

The reporting unit is configured to record a validated status uponreception of a validation instruction provided by the operator. Thisvalidation instruction comprises preferably an oral instruction and/or aspecific displacement of a hand of the operator, for instance adisplacement marking the intention of the operator to tap the validationmark 112.

The reporting unit is configured to record a defective status uponreception of a defect report provided by the operator. This defectreport comprises preferably an oral instruction and/or a specificdisplacement of a hand of the operator, for instance a displacementmarking the intention of the operator to tap the defect mark 114.

The reporting unit is further configured to record, for each defectivestatus, a cause of defect. This cause of defect is recorded uponreception of an information from the operator designating one designatedcause among the defect causes listed by the defects menu 136, thisinformation being for instance in the form of an oral phrase and/or of aspecific displacement of a hand of the operator, for instance adisplacement marking the intention of the operator to tap the defectedcause in the defect menu 136.

Besides, the reporting unit is configured to provide the recordedstatuses to the display management unit 34 and to the server 16. Thereporting unit is configured to provide the recorded causes of defect tothe server 16.

An assistance method 200 for assisting an operator in performing aquality control on the product 12, using the assistance system 10, willnow be described, with reference to FIGS. 8 and 9.

This assistance method 200 comprises firstly a configuration step 210.

During this configuration step 210, the head-mounted display device 14is brought close to the product 12 by the operator, so as to place anidentification code 140 (FIG. 3) associated to the product 12, forinstance placed on said product 12, in the field of view of the standardcamera 56. As a result thereof, the code reader reads the identificationcode 140, and decodes it into an identifier which is provided to theconfiguration unit, during a first substep 212. Then, during a secondsubstep 214, the configuration unit retrieves in the memory 64 of theserver 16, using the identifier decoded by the code reader, the virtualmockup, the initial relative positioning information, and the list ofsubtasks which are, in said memory 64, associated to said identifier,and provides the virtual mockup to the display management unit 34,together with the list of subtasks, the initial relative positioninginformation, and, for each subtask of the list, a part information, anaction information and, optionally, at least one of a picture and ananimation corresponding to said subtask

The step 210 is followed by an initialization step 220.

This initialization step 220 comprises a first substep 222 of equippingthe support 18 on the product 12, followed by a second substep 224 ofplacing the head-mounted display device 14 in the slot 74 of the support18, and then by a third substep 226 of informing the head-mounteddisplay device 14 that this placement has been done, for instance by theoperator pushing a button on the head-mounted display device 14 orperforming a predetermined gesture in the field of view of one of thecameras 50, 52, 56 of the head-mounted display device 14. As a resultthereof, the displacement tracking unit 36 sets the current estimationof the position and of the orientation of the head-mounted displaydevice 14 in real-world space as the reference position and thereference orientation during a fourth substep 228, and starts trackingdisplacement of the head-mounted display device 14 relatively to thesereference position and orientation.

The initialization step 220 is followed by a step 230 of wearing thehead-mounted display device 14, during which the operator mounts thehead-mounted display device 14 upon his head. This wearing step isfollowed, optionally, by a calibration step 240, during which theinterpapillary distance of the operator is recorded by the displaymanagement unit 34 so as to adapt the images displayed by the displaymanagement unit 34 to said interpapillary distance.

Further to the wearing step 230 and, where appropriate, to thecalibration step 240, the method 200 comprises an assistance step 250during which the system 10 assists the operator in accomplishing asubtask of the task, for which the operator has to perform an action ona monitored part of the product 12.

At the first instance of this assistance step 250, this subtask consistsof the first subtask mentioned in the list retrieved by theconfiguration unit.

Here, at the beginning of the first instance of this step 250, theoperator is at a distance of the monitored part above the thresholddistance and looks at the product 12, so that the operator views a firstview of a scene including the product 12. He is however placed in such aposition that he has no direct view on the monitored part. Thecorresponding first view of the scene in shown in FIG. 3.

This assistance step 250 comprises a first substep 251 of displayinggeneral assistance images. This substep 251 comprises the display, bythe display management unit 34, of the task advancement image 116,superimposed on the view of the scene, as a carrousel above the product12.

The assistance step 250 also comprises a second substep 252,substantially simultaneous with the first substep 251, of displayingrelevant assistance images for the execution of the subtask. During thissecond substep 252, the display management unit 34 displays the guidanceimages 104 and 106 to guide the operator toward a position in which hewill have a direct view on the monitored part. The display managementunit 34 also displays the highlighting images 100, 102, the action image108, the validation mark 112 and the defect mark 114 superimposed on theview of the scene. Alternatively, the images 100, 102, 108 and the marks112 and 114 are displayed only once the operator has a direct view onthe monitored part.

During this second substep 252, the display management unit 34 deducesfrom the initial relative positioning information, and from thedisplacement tracked by the displacement tracking unit 36 since theinitialization step 220, estimated current relative position andorientation of the product 12 relatively to the view source, and adjuststhe images 100, . . . , 108, 112, 114 to said estimated current relativeposition and orientation.

Following the first and second substeps 251, 252 is a displacementsubstep 253 during which the operator moves toward the area highlightedby the square 104 until he gets a second view of the scene in which hehas a direct view on the monitored part. This second view is shown inFIG. 4.

During this displacement substep 253, the display management unit 34updates the calculation of the estimated position and orientation at thepredetermined frequency, and refreshes the display of the images 100, .. . , 108, 112, 114 at the same frequency so as to continuously adjustsaid images 100, . . . , 108, 112, 114 to said estimated relativeposition and orientation. Once the operator has reached a position inwhich he has a direct view on the monitored part, the display managementunit 34 refreshes the display of the images 100, . . . , 108, 112, 114so as to remove the guidance images 104, 106.

Following the substep 253 is a step 254 of the operator requestingdisplay of additional information, such as a plan of the product 12,followed by a step 255 of the display management unit 34 displaying in aframe 110 an image providing said additional information.

Then, the operator approaches the monitored part during a substep 256,until a position in which the eyes of the operator are at a distancefrom the monitored part under the threshold distance, and the operatorviews a third view of the scene which is shown in FIG. 5.

During this substep 256, the distance of the view source from themonitored part crosses the threshold distance. As a consequence thereof,the display of the task advancement image 116 is changed from thecarrousel shape to the shape shown in FIG. 5, in which it is displayedas a succession of icons adjacent to the ring 100.

Also, during this substep, the display management unit 34 keeps updatingthe calculation of the estimated position and orientation, andrefreshing the display of the images 100, . . . , 108, 112, 114.

Next, during a substep 257, the operator performs the action shown inthe shield 108. If the operation involves manipulating the monitoredpart, the operator merely needs to try grabbing the virtualrepresentation 102 thereof, his hands then naturally falling on themonitored part.

Following step 250, the method 200 comprises a step 260 of reporting theresults of the control performed on the monitored part.

This step 260 comprises a first substep 261, during which the operatorprovides the reporting unit with either a validation instruction or adefect instruction, followed by a substep 262 of the reporting unitupdating correspondingly the status of the subtask into a validatedstatus or a defective status.

If the updated status is a defective status, the substep 262 is followedby a substep 263 of the operator providing, a cause of the defect to thereporting unit, followed itself by a substep 264 of recording this causeof defect, further followed by a transmission substep 265. If thereported instruction is a validation instruction, the substep 262 isfollowed directly by the transmission substep 265.

During the transmission substep 265, the status of the subtask istransmitted by the reporting unit to the server 16 and to the displaymanagement unit 34. In case the transmitted status is a defectivestatus, transmission to the server 16 is accompanied with thetransmission of the defect cause.

Transmission substep 265 is followed by a substep 266 of the displaymanagement unit 34 updating the task advancement image 116 in line withthe transmitted status.

Step 260 is followed by a step 270 of checking if the executed subtaskwas the last task of the list or not. If it is the case, the method ends200, and the head-mounted display device 14 is placed in an awaitingstep. If the executed subtask was not the last task of the list, thensteps 250, 260 and 270 are repeated with the following subtask mentionedin the list. It should be noted that, when step 250 is repeated, theoperator may now be placed in a position in which he has direct view onthe monitored step, in which case the second substep 252 would notcomprise display of the guidance images 104 and 106.

Thanks to the invention described above, the operator is able to focuson the right quality controls depending to the variant he has tocontrol. Furthermore, since the operator is guided from subtask to thefollowing subtask, he cannot miss a control. As a result thereof, therisk of errors is significantly reduced.

Furthermore, the operator is able to perform his controls free-handed:the quality control is therefore facilitated.

Moreover, as the control channel is fully digitalized from end to end,the costs and activities related to paper printing in sequence areremoved, and it becomes possible to use real time quality keyperformance indicators that will enable to react quickly to fix therecurrent errors which may appear in the assembly line.

It should be noted that, even though the example above is provided forthe specific case in which the front-end display device 14 consists of ahead-up display device, the invention is not limited to this case. As analternative (not shown), the front-end display device 14 consists of atablet, such as a tablet implementing the Google® Tango® technology, theview source the being a camera of the tablet and the view of the scenethen being the view captured by the camera and reproduced on a screen ofthe tablet.

It should also be noted that alternatives are also available for theinitialization step 220. Instead of the partly automatized stepdescribed above, the step 220 may consist of a fully manual step,comprising the following substeps:

-   -   display, by the display management unit 34, of an image of the        entire virtual mockup of the product 12,    -   displacement of the operator in the scene, until he reaches a        position and an orientation in which the image of the virtual        mockup matches the product 12, or command, by the operator, of        displacement of the image of the virtual mockup the his view of        the scene, until the image of the virtual mockup reaches a        position in which it matches the product 12, and    -   validation, by the user, that a correct positioning of the        virtual mockup in the scene has been reached.

The invention claimed is:
 1. An assistance method for assisting anoperator in performing a task on a product, the method comprising a stepof displaying, superimposed on a view of a scene including a monitoredpart of the product on which a subtask of the task has to be performed,at least one assistance image intended to assist the operator inperforming this subtask, characterized in that said at least oneassistance image includes at least one highlighting image highlightingthe monitored part, wherein the view of the scene is provided by a viewsource, and the method further comprises the following steps: tracking adisplacement of the view source from a first viewing position in thescene, in which the product occupies a first relative position with afirst relative orientation relatively to the view source, toward asecond viewing position in which the product occupies a second relativeposition with a second relative orientation relatively to the viewsource, calculating an estimation of the second relative position and ofthe second relative orientation from the first relative position andorientation and from the tracked displacement, and refreshing thedisplay of the at least one assistance image so as to adjust said atleast one assistance image to the estimation of the second relativeposition and orientation, wherein the first viewing position consists ofan initial viewing position in which the first relative position andorientation consist of predetermined relative position and orientationknown by a display management unit in charge of displaying the at leastone assistance image, wherein the method further comprises a step ofequipping the product with a support comprising a slot able forreceiving the view source and adapted so that, when the support isequipped on the product and the view source is received in said slot,the view source is placed in said initial viewing position, followed bya step of placing the view source in the slot of said support.
 2. Theassistance method of claim 1, wherein the at least one highlightingimage comprises a ring surrounding the monitored part and preferablycentered on said monitored part, and/or a virtual representation of saidmonitored part, said virtual representation being positioned so as tomatch the monitored part in the view of the scene.
 3. The assistancemethod of claim 1, wherein the at least one assistance image isdisplayed on at least one display surface.
 4. The assistance method ofclaim 3, wherein the view of the scene consists of a view of the scenethrough the at least one display surface.
 5. The assistance method ofclaim 3, wherein the at least one display surface is part of ahead-mounted display device, said head-mounted display device beingadapted to be worn by the operator in such a manner that the displaysurface faces a respective one of his eyes.
 6. The assistance method ofclaim 5, wherein the at least one display surface includes a primarydisplay surface and a secondary display surface, the at least oneassistance image consisting of a stereoscopic image including a primaryimage displayed on the primary display surface and a secondary imagedisplayed on the secondary display surface, said primary and secondaryimages consisting of images of a same element viewed from slightlydifferent angles.
 7. The assistance method of claim 5, wherein the viewsource comprises the position intended to be occupied by at least one ofthe eyes of the operator wearing the head-mounted display device.
 8. Theassistance method of claim 1, wherein the at least one assistance imagecomprises at least one guidance image highlighting an area where toplace the view source so as to provide a better view on the monitoredpart.
 9. The assistance method of claim 1, comprising a step ofdisplacing the view source in the scene in manner that the distance ofthe view source from the monitored part crosses a threshold distance,followed by a step of modifying the position of at least part of the atleast one assistance image.
 10. The assistance method of claim 1,wherein the task to be performed comprises a quality control of theproduct.
 11. The assistance method of claim 1, wherein the productcomprises a part of an automotive vehicle.
 12. The assistance method ofclaim 1, comprising, prior to displaying the at least one assistanceimage, a step followed by a step of, using said identification code,automatically retrieving a list of at least one subtask to be performedon the product, the at least one assistance image being intended toassist the operator in performing a subtask of said list.
 13. Anassistance system for assisting an operator in performing a task on aproduct, the assistance system comprising a display management unitconfigured for displaying, superimposed on a view of a scene including amonitored part of the product on which a subtask of the task has to beperformed, at least one assistance image intended to assist the operatorin performing this subtask, characterized in that said at least oneassistance image includes at least one highlighting image highlightingthe monitored part, wherein the assistance system includes a viewsource, at least one sensor to track a displacement of the view sourcefrom a first viewing position in the scene, in which the productoccupies a first relative position and a first relative orientationrelatively to the view source, toward a second viewing position in whichthe product occupies a second relative position and a second orientationrelatively to the view source, and the display management unit isconfigured to calculate an estimation of the second relative positionand orientation from the first relative position and orientation andfrom the tracked displacement, and to refresh the display of the atleast one assistance image so as to adjust said at least one assistanceimage to the estimation of the second relative position and orientation,wherein the first viewing position consists of an initial viewingposition in which the first relative position and orientation consist ofpredetermined relative position and orientation known by the displaymanagement unit, the system further comprising a support able to bemounted on the product and including a slot able for receiving the viewsource, said support being adapted so that, when it is mounted on theproduct and the view source is received in said slot, the view source isplaced in the initial viewing position.
 14. The assistance system ofclaim 13, wherein the at least one highlighting image comprises a ringsurrounding the monitored part and preferably centered on said monitoredpart, and/or a virtual representation of said monitored part, saidvirtual representation being positioned so as to match the monitoredpart in the view of the scene.
 15. The assistance system of claim 13,comprising at least one display surface, the display management unitbeing configured to display the at least one assistance image on said atleast one display surface.
 16. The assistance system of claim 15,wherein the view of the scene consists of a view of the scene throughthe at least one display surface.
 17. The assistance system of claim 14,comprising a head-mounted display device which includes the at least onedisplay surface, said head-mounted display device being adapted to beworn by the operator in such a manner that the display surface faces arespective one of his eyes.
 18. The assistance system of claim 17,wherein the at least one display surface comprises a primary displaysurface and a secondary display surface, and the display management unitis configured for displaying a primary image on the primary displaysurface and a secondary image on the secondary display surface, saidprimary and secondary images consisting of images of a same elementviewed from slightly different angles, so as to form the at least oneassistance image as a stereoscopic image resulting from the combination,by the eyes of the operator, of the primary and secondary images. 19.The assistance system claim 13, wherein the view source comprises theposition intended to be occupied by at least one of the eyes of theoperator wearing the head-mounted display device.
 20. The assistancesystem of claim 13, wherein the at least one assistance image comprisesat least one guidance image highlighting an area where to place the viewsource so as to provide a better view on the monitored part.
 21. Theassistance system of claim 13, wherein the display management unit isconfigured for modifying the position of at least part of the at leastone assistance image when the distance of the view source from themonitored part crosses a threshold distance.
 22. The assistance systemof claim 13, wherein the task to be performed comprises a qualitycontrol of the product.
 23. The assistance system of claim 13, whereinthe product comprises a part of an automotive vehicle.
 24. Theassistance system of claim 13, comprising a memory storing a virtualmockup of the product.
 25. The assistance system of claim 13, comprisinga memory storing a first of at least one subtask to be performed, areader able to read an identification code worn by the product, and aconfiguration unit configured to retrieve said list in the memory, usingthe identification code read by the reader, and the display managementunit being configured to display the at least one assistance imagedepending on the relevant list retrieved from the memory.